Skip to content

[opt](paimon) support variant access path in paimon jni reader - #66547

Merged
yiguolei merged 2 commits into
apache:branch-4.1from
zhangstar333:branch-4.1-variant-sub
Aug 7, 2026
Merged

[opt](paimon) support variant access path in paimon jni reader#66547
yiguolei merged 2 commits into
apache:branch-4.1from
zhangstar333:branch-4.1-variant-sub

Conversation

@zhangstar333

@zhangstar333 zhangstar333 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Problem Summary:

support read variant sub path in jni reader using access path

|   0:VPAIMON_SCAN_NODE(87)                                                     |
|      table: test_variant.variant_db.variant_shredded                          |
|      inputSplitNum=1, totalFileSize=0, scanRanges=1                           |
|      partition=1/0                                                            |
|      cardinality=2, numNodes=1                                                |
|      nested columns:                                                          |
|        payload:                                                               |
|          origin type: variant                                                 |
|          all access paths: [payload.name]                                     |
|      pushdown agg=NONE                                                        |
|      paimonNativeReadSplits=0/1                                               |
|      predicatesFromPaimon: NONE                                               |
|      final projections: id[#0], CAST(element_at(payload[#1], 'name') AS text) |
|      final project output tuple id: 1

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@zhangstar333
zhangstar333 requested a review from yiguolei as a code owner August 6, 2026 10:28
@hello-stephen

hello-stephen commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Gabriel39

Gabriel39 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Reviewed at b148688 against the latest branch-4.1. I did not find a blocking correctness issue. The three-way merge also preserves the recently merged fix that unsets parquet.variant.shreddingSchema before writing unshredded files.

Non-blocking test coverage suggestions:

  • Add an end-to-end JNI regression for a nested shredded object path such as profile.city. Nested extraction is currently covered only by Java unit tests.
  • Add a JNI regression for an unsupported numeric or array path such as payload[tags][0], verifying that it correctly falls back to reading the full Variant.
  • Add feature-gate tests for Variant nested inside ARRAY, MAP, and STRUCT; the implementation checks these types recursively, while the current test covers only a top-level Variant.

@Gabriel39

Gabriel39 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

run buildall

1 similar comment
@zhangstar333

zhangstar333 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

hello-stephen commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 93.75% (15/16) 🎉
Increment coverage report
Complete coverage report

@zhangstar333
zhangstar333 force-pushed the branch-4.1-variant-sub branch from b148688 to 64634e9 Compare August 7, 2026 03:50
@zhangstar333

zhangstar333 commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

hello-stephen commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 54.55% (6/11) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

hello-stephen commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 100.00% (22/22) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 58.33% (24737/42412)
Line Coverage 42.60% (249476/585579)
Region Coverage 38.64% (197085/510025)
Branch Coverage 40.06% (90280/225384)

@yiguolei
yiguolei merged commit 6a47008 into apache:branch-4.1 Aug 7, 2026
27 of 32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei yiguolei approved these changes

Assignees

No one assigned

Labels

dev/5.0.x

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@zhangstar333 @hello-stephen @Gabriel39 @yiguolei